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1.  Statement  of  the  Problem  Studied 

The  area  of  this  research  is  the  numerical  solution  of  nonlinear  optimization  problems.  Optimiza¬ 
tion  problems  are  one  of  the  main  forms  of  mathematical  problems  whose  computer  solution  is  desired  by 
scientists  and  engineers.  The  need  to  solve  such  problems  arises  in  a  huge  variety  of  computer  simula¬ 
tions  and  data  analyses,  with  examples  including  optimal  design  of  aircraft  and  spacecraft  for  low  cost  or 
high  efficiency,  optimal  control  of  chemical  processes,  and  calculation  of  the  native  states  of  biological 
molecules.  In  recent  years,  as  computer  power  has  increased,  the  feasibility  of  investigating  optimal 
designs  has  increased,  and  along  with  this  the  need  for  improved  optimization  methods  has  grown.  This 
requires  fundamental  research  in  optimization  algorithms,  especially  for  classes  of  large-scale  problems. 
In  addition,  much  of  this  computation  must  utilize  the  fastest  available  computers,  which  increasingly  are 
parallel  computers.  Thus  the  development  of  optimization  methods  that  are  well-suited  to  parallel  com¬ 
puters  is  also  a  pressing  need. 

One  of  the  main  emphases  of  this  research  program  in  recent  years  has  been  the  development  of 
global  optimization  methods  for  the  solution  of  large  molecular  configuration  problems.  Global  optimiza¬ 
tion  means  finding  the  lowest  minimizer  of  a  nonlinear  function  that  may  have  numerous  local  minimiz- 
ers.  The  objective  of  this  research  is  to  develop  methods  that  are  reliably  capable  of  solving  difficult, 
large  nonlinear  global  optimization  problems.  Our  current  work  is  investigating  methods  to  find  the 
native  configurations  of  proteins  and  polymers.  This  is  a  problem  of  great  importance  in  science;  it 
includes  the  well-known  protein  folding  problem  as  well  as  the  investigation  of  polymers  that  are  used  to 
make  new  materials.  The  resultant  optimization  problems  are  very  difficult  because  they  have  extremely 
large  numbers  of  local  minimizers  that  have  similar  function  values. 

Another  main  focus  of  our  research  is  the  development  of  new  algorithms  for  large-scale  uncon¬ 
strained  and  constrained  optimization  problems,  including  limited-memory  methods  for  problems  with 
many  thousands  of  variables,  and  interior  point  methods  for  nonlinearly  constrained  problems.  We  are 
also  investigating  theoretical  convergence  issues  for  optimization  algorithms  that  have  practical  conse¬ 
quences  in  how  problems  are  formulated,  and  new  approaches  to  one  of  the  key  numerical  linear  algebra 
subproblems  that  is  particular  to  optimization  algorithms. 
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2.  Summary  of  the  Most  Important  Results 

The  most  important  contribution  under  this  grant  has  been  a  vast  improvement  in  the  ability  of 
large-scale  global  optimization  methods  to  solve  very  difficult  molecular  configuration  problems.  The 
improvements  are  shown  both  in  the  size  and  in  the  complexity  of  the  problems  that  can  now  be  solved. 
At  the  beginning  of  this  research  period,  we  were  just  beginning  to  solve  molecular  cluster  problems  with 
up  to  hundreds  of  parameters.  By  the  end  of  the  research  period,  we  had  progressed  from  these  cluster 
problems  to  much  more  difficult  and  important  protein  folding  problems  with  thousands  of  variables. 

The  most  fundamental  research  advance  that  contributed  to  the  improvements  in  our  large-scale 
global  optimization  algorithms  was  the  development  and  understanding  of  new  smoothing  approaches. 
Smoothing  refers  to  approaches  that  replace  the  .highly  nonlinear  objective  function  by  one  that  retains  its 
coarse  grain  features  but  reduce  its  fine  grain  variations.-  The  basic  objective  of  smoothing  is  to  first  solve 
an  easier  computational  problem  using  the  smoothed  function  and  then  use  this  to  find  the  solution  of  the 
original  problem.  Smoothing  has  been  applied  to. global  optimization  problems  from  molecular  chemistry 
before,  but  involved  the  use  of  expensive  convolution-based  approaches  that  also  required  ah-hoc  approxi¬ 
mations  to  eliminate  poles.  One  main  contribution  of  our  research  was  the  introduction  a  simple,  cheap, 
analytic  method  of  smoothing  molecular  chemistry  objective  functions,  and  demonstration  that  it  per¬ 
forms  as  well  as  previous,  expensive  smoothing  approaches.  Careful  experiments  showed  that  our  simpler 
and  cheaper  function  was  qualitatively  very  similar  in  its  form  and  its  properties  as  the  functions  that 
result  from  convolution.  A  second  contribution  was  been  to  demonstrate,  via  experiments,  some  funda¬ 
mental  limitations  of  smoothing  approaches  for  global  optimization  that  may  not  have  been  recognized 
previously.  In  particular,  we  showed  that  for  a  variety  of  problems,  smoothing  introduces  very  significant 
“order  flips”,  meaning  that  the  order  of  the  smoothed  minimizers  does  not  correspond  to  the  order  of  the 
nonsmoothed  minimizers  from  which  they  descend.  This  research  showed  that  it  is  not  sufficient  to  com¬ 
bine  smoothing  with  local  optimization  as  had  previously  been  done;  rather  global  optimization  tech¬ 
niques  must  be  used  at  various  levels  of  smoothing. 

Based  on  this  understanding,  we  developed  an  algorithmic  framework  that  combines  smoothing 
with  the  Pi’s  stochastic -perturbation  global  optimization  approach  for  large-scale  global  optimization.  We 
also  made  significant  refinements  to  the  stochastic-perturbation  approach  to  permit  it  to  apply  to  protein 
problems  as  opposed  to  molecular  clusters.  These  particularly  included  the  development  of  new  heuris¬ 
tics  to  select  the  parameters  which  are  used  in  the  crucial  small  scale  global  optimization  phase  of  our 
method,  as  well  as  entirely  new  methods  to  generate  initial  structures  for  our  method.  The  combination  of 
the  two  approaches,  the  modified  stochastic-perturbation  method  and  smoothing,  produced  very  promis¬ 
ing  results.  Most  importantly,  we  were  able  to  find  optimal  configurations  of  the  protein  polyalanine  for 
20,  30,  40  and  58  amino  acids  (with  approximately  30  variables  per  amino  acid).  In  the  first  three  cases, 
the  optimal  structure  is  alpha-helical,  and  could  be  found  by  our  algorithm  with  or  without  smoothing,  but 
was  found  far  more  readily  with  smoothing  for  the  large  protein.  In  the  last  case,  the  optimal  solution 
consists  of  two  alpha-helical  segments  connected  with  a  turn  in  the  rough  overall  shape  of  the  character 
alpha.  While  chemists  had  shown  simply  by  ' generating  configurations  that  this  was  a  lower  energy  struc¬ 
ture  than  a  straight  alpha  helix,  we  believe  that  ours  is  the  first  work  to  find  this  shape  via  optimization 
techniques.  This  shows  the  promise  of  our  approach  for  finding  the  complex  shapes  of  more  complicated 
proteins.  Smoothing  proved  crucial  in  both  the  efficiency  and  the  quality  of  our  work  on  this  larger  pro¬ 
tein.  Our  ongoing  research  is  now  considering  more  difficult  proteins  and  further  algorithmic 
improvements. 
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As  part  of  this  research,  we  began  utilizing  “biasing”  techniques  in  our  generation  of  initial  struc¬ 
tures.  This  work,  conducted  in  conjunction  with  the  group  of  Teresa-Head  Gordon  of  Lawrence  Berkeley 
Laboratory,  is  based  upon  the  antiion  approach  that  she  developed  with  F.  Stillinger  of  Bell  Labs.  From 
an  optimization  viewpoint,  it  uses  penalty  terms  to  reward  secondary  structure  that  has  been  predicted  by 
techniques  such  as  neural  nets.  Experiments  with  polyalanine  showed  that  biasing  has  the  potential  to 
greatly  improve  our  initial  structures,  and  that  it  should  be  an  important  part  of  techniques  for  more  com¬ 
plex  proteins. 

A  related  advance  was  the  application  of  our  global  optimization  methodology  to  a  quite  different 
class  of  problems,  distance  geometry  problems,  that  also  arise  in  molecular  chemistry.  Distance  geometry 
problems  also  are  global  optimization  problems  that  arise  from  trying  to  find  the  configuration  that  comes 
closest  to  matching  a  set  of  approximate  pairwise  distances  between  some  subset  of  the  pairs  of  atoms  in 
a  molecule.  Our  stochastic  perturbation  approach  was  significantly  more  successful  than  previously  tried 
optimization  approaches  in  solving  test  problems  with  up  to  2000  variables.  This  research  demonstrates 
the  versatility  of  our  global  optimization  approach. 

Earlier  in  the  research  period  we  applied  our  stochastic-perturbation  global  optimization  approach 
to  molecular  cluster  problems  with  excellent  results.  We  completed  the  testing  of  our  methods  on  two 
well-known  molecular  cluster  problems,  Lennard-Jones  clusters  and  water.  On  the  much-studied 
Lennard-Jones  problems,  our  method  produced  better  results  than  any  known  general  or  special  purpose 
method,  for  problems  with  up  to  76  atoms  (228  parameters).  This  included  finding  the  best  known  mini- 
mizers  for  all  of  these  problems,  a  new  lower  minimizer  for  the  75  atom  problem,  and  recently  discovered 
lower  minimizers  for  the  66  and  72  atom  problems.  The  method  also  found  far  lower  energy  configura¬ 
tions  for  clusters  of  water  molecules  with  up  to  288  parameters  than  a  previous  method  developed  by 
chemists. 

In  our  research  on  limited  memory  methods  for  large  scale  problems,  we  developed  high  quality 
software  for  our  new  approach  for  solving  large  nonlinear  optimization  problems  subject  to  simple 
bounds  on  the  variables.  This  software  is  intended  for  problems  in  which  information  on  the  Hessian 
matrix  is  difficult  to  obtain,  or  for  large  dense  problems. 

Our  research  in  large  scale  constrained  optimization  concentrated  on  the  development  of  interior 
point  approaches  for  nonlinearly  constrained  problems.  This  work  has  resulted  in  a  new  primal  method 
with  strong  global  convergence  properties.  Additionally,  experiments  with  primal  and  primal-dual  ver¬ 
sions  of  this  method  have  demonstrated  promising  computational  performance  on  large  scale  problems. 
Ongoing  research  appears  to  be  establishing  this  approach  as  one  of  the  leading  ones  for  large  scale  con¬ 
strained  optimization.  . 

In  the  convergence  analysis  of  singular  constrained  optimization  problems,  we  developed  a  new  set 
of  optimality  conditions  for  singular  equality  constrained  optimization  problems.  These  appear  to  be  the 
first  conditions  that  deal  effectively  with  the  important  practical  case  where  the  Jacobian  matrix  of  the 
constraints  is  singular  at  the  solution  and  the  Lagrange  multipliers  are  unbounded.  We  can'  show  that  our 
new  optimality  condition  is  a  necessary  condition  for  singular  constrained  minimizers  satisfying  rather 
general  conditions,  as  well  as  being  satisfied  for  a  broad  set  of  problems. 

Another  significant  contribution  has  been  an  improvement  to  our  modified  Cholesky  factorization 
algorithm.  The  modified  Cholesky  factorization  is  an  essential  element  of  many  unconstrained  and  con¬ 
strained  optimization  approaches.  Our  new  algorithm,  introduced  several  years  ago,  has  become  widely 
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used,  but  recently  some  cases  had  arisen  where  its  performance  was  not  good,  especially  in  interior  point 
approaches  to  constrained  optimization.  A  fairly  simple  modification  to  the  factorization  has  enabled  it  to 
handle  all  known  problematic  cases  excellently,  while  still  retaining  its  previous  good  computational  and 
theoretical  properties. 

Finally,  we  completed  research  on  several  topics  in  tensor  methods,  for  large  scale  and  constrained 
optimization.  We  also  completed  and  released  software  for  using  tensor  methods  for  solving  systems  on 
nonlinear  equations  and  nonlinear  least  squares  problems.  This  software  has  been  used  extensively  and 
appears  to  be  effective  in  enabling  users  to  solve  some  problems  that  they  couldn’t  solve  before,  or  to 
solve  problems  more  efficiently. 
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